Developing an AI application

Going forward, AI algorithms will be incorporated into more and more everyday applications. For example, you might want to include an image classifier in a smart phone app. To do this, you'd use a deep learning model trained on hundreds of thousands of images as part of the overall application architecture. A large part of software development in the future will be using these types of models as common parts of applications.

In this project, you'll train an image classifier to recognize different species of flowers. You can imagine using something like this in a phone app that tells you the name of the flower your camera is looking at. In practice you'd train this classifier, then export it for use in your application. We'll be using this dataset of 102 flower categories, you can see a few examples below.

The project is broken down into multiple steps:

  • Load and preprocess the image dataset
  • Train the image classifier on your dataset
  • Use the trained classifier to predict image content

We'll lead you through each part which you'll implement in Python.

When you've completed this project, you'll have an application that can be trained on any set of labeled images. Here your network will be learning about flowers and end up as a command line application. But, what you do with your new skills depends on your imagination and effort in building a dataset. For example, imagine an app where you take a picture of a car, it tells you what the make and model is, then looks up information about it. Go build your own dataset and make something new.

First up is importing the packages you'll need. It's good practice to keep all the imports at the beginning of your code. As you work through this notebook and find you need to import a package, make sure to add the import up here.

Please make sure if you are running this notebook in the workspace that you have chosen GPU rather than CPU mode.

In [1]:
# Import Modules
%matplotlib inline


from __future__ import print_function, division

import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim import lr_scheduler
from torch.autograd import Variable
import numpy as np
import torchvision
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import time
import os
import copy
from PIL import Image
import torch.nn.functional as F
import json
plt.ion()

use_gpu = torch.cuda.is_available()
if use_gpu:
  print("Using CUDA")
Using CUDA

Load the data

Here you'll use torchvision to load the data (documentation). The data should be included alongside this notebook, otherwise you can download it here. The dataset is split into three parts, training, validation, and testing. For the training, you'll want to apply transformations such as random scaling, cropping, and flipping. This will help the network generalize leading to better performance. You'll also need to make sure the input data is resized to 224x224 pixels as required by the pre-trained networks.

The validation and testing sets are used to measure the model's performance on data it hasn't seen yet. For this you don't want any scaling or rotation transformations, but you'll need to resize then crop the images to the appropriate size.

The pre-trained networks you'll use were trained on the ImageNet dataset where each color channel was normalized separately. For all three sets you'll need to normalize the means and standard deviations of the images to what the network expects. For the means, it's [0.485, 0.456, 0.406] and for the standard deviations [0.229, 0.224, 0.225], calculated from the ImageNet images. These values will shift each color channel to be centered at 0 and range from -1 to 1.

In [2]:
data_dir = 'flowers'
TRAIN = 'train'
VAL = 'valid'
TEST = 'test'
In [3]:
# TODO: Define your transforms for the training, validation, and testing sets
data_transforms = {
    TRAIN: transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomRotation(30),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])
                                    ]), 
    VAL: transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])]),
    TEST: transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])
    ])                            
}

# TODO: Load the datasets with ImageFolder
image_datasets = { 
    x: datasets.ImageFolder(
        os.path.join(data_dir, x),
        transform=data_transforms[x]
    )
    for x in [TRAIN, VAL, TEST]
}    
    
# TODO: Using the image datasets and the trainforms, define the dataloaders
dataloaders = { 
    x: torch.utils.data.DataLoader(image_datasets[x], batch_size=32, shuffle=True)
    for x in [TRAIN, VAL, TEST]
}

# Provide dataset description

dataset_sizes = {x: len(image_datasets[x]) for x in [TRAIN, VAL, TEST]}

for x in [TRAIN, VAL, TEST]:
    print("{} images under {}".format(dataset_sizes[x], x))

print("Classes: ")
class_names = image_datasets[TRAIN].classes
print(image_datasets[TRAIN].classes)
6552 images under train
818 images under valid
819 images under test
Classes: 
['1', '10', '100', '101', '102', '11', '12', '13', '14', '15', '16', '17', '18', '19', '2', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '3', '30', '31', '32', '33', '34', '35', '36', '37', '38', '39', '4', '40', '41', '42', '43', '44', '45', '46', '47', '48', '49', '5', '50', '51', '52', '53', '54', '55', '56', '57', '58', '59', '6', '60', '61', '62', '63', '64', '65', '66', '67', '68', '69', '7', '70', '71', '72', '73', '74', '75', '76', '77', '78', '79', '8', '80', '81', '82', '83', '84', '85', '86', '87', '88', '89', '9', '90', '91', '92', '93', '94', '95', '96', '97', '98', '99']

Label mapping

You'll also need to load in a mapping from category label to category name. You can find this in the file cat_to_name.json. It's a JSON object which you can read in with the json module. This will give you a dictionary mapping the integer encoded categories to the actual names of the flowers.

In [4]:
with open('cat_to_name.json', 'r') as f:
    cat_to_name = json.load(f)
In [5]:
# utility functions
# show image in matplotlib
def imshow(inp, title=None):
    inp = inp.numpy().transpose((1,2,0))
    mean = np.array([0.485, 0.456, 0.406])
    std = np.array([0.229, 0.224, 0.225])
    inp = std * inp + mean
    inp = np.clip(inp, 0, 1)
    plt.figure(figsize=(20,20))
    plt.axis('off')
    plt.imshow(inp)
    if title is not None:
        plt.title(title)
    plt.pause(0.001)
    
# map model classes to category names from json file for lists
def classes_not_numbers_list(classes, cat_to_name):
    class_map = []
    # run through each class
    for c in classes:
        if c == '0':
              class_map.append('pink primrose')
        for key, value in cat_to_name.items():
            if c == key:
                class_map.append(value)
    
    return class_map
  
# display databatch
def show_databatch(inputs, classes):
    out = torchvision.utils.make_grid(inputs)
    imshow(out, title=classes_not_numbers_list([class_names[x] for x in classes], cat_to_name))

def show_databatch_dev(inputs, classes):
    out = torchvision.utils.make_grid(inputs)
    imshow(out, title=[class_names[x] for x in classes])
    
# map model classes to category names from json file for tensors
def classes_not_numbers_tensor(classes, cat_to_name):
    class_map = []
    for c in classes.numpy().tolist():
        for key, value in cat_to_name.items():
            if c == key:
                class_map.append(value)
    return class_map
In [6]:
# get and display some data
inputs, classes = next(iter(dataloaders[TRAIN]))

show_databatch(inputs, classes)
In [7]:
# Visualize Model
def visualize_model(model, num_images=32):
    was_training = model.training
    
    # Set model for evaluation
    model.eval()
    images_count = 0
    fig = plt.figure(figsize=(10,10))
    
    with torch.no_grad():
        for i, data in enumerate(dataloaders[VAL]):
            inputs, labels = data
            size = inputs.size()[0]

            if use_gpu:
                model.to('cuda')
                inputs, labels = inputs.to('cuda'), labels.to('cuda')

            outputs = model(inputs)
            _, preds = torch.max(outputs, 1)
            
            predicted_labels = [preds[j] for j in range(inputs.size()[0])]
            
            print('Ground Truth:')
            show_databatch(inputs.data.cpu(), labels.data.cpu())
            print('Prediction:')
            show_databatch(inputs.data.cpu(), predicted_labels)

            del inputs, labels, outputs, preds, predicted_labels
            torch.cuda.empty_cache()
            
            images_count += size
            
            
            if images_count >= num_images:
                model.train(mode=was_training)
                return
        model.train(mode=was_training) # Revert model back to training state
In [13]:
def train_model(model, critereon, optimizer, scheduler, num_epochs=10):
    since = time.time()
    
    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0
    

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' *10)

        # Each epoch has a training and a validation phase
        for phase in [TRAIN, VAL]:
            if phase == TRAIN:
                scheduler.step()
                model.train() # set model to training mode
            else:
                model.eval()
            
            running_loss = 0.0
            running_corrects = 0
            steps = 0
            for inputs, labels in dataloaders[phase]:
                steps += 1
                if use_gpu:
                    model.to('cuda')
                    inputs, labels = inputs.to('cuda'), labels.to('cuda')
                    
                #zero the parameter gradients
                optimizer.zero_grad()
                
                # forward pass
                # track history if in train
                with torch.set_grad_enabled(phase == TRAIN):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = critereon(outputs, labels)
                    
                    # backward pass + optimize if in training phase
                    if phase == TRAIN:
                        loss.backward()
                        optimizer.step()
                        
                # update stats
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)
                
                epoch_loss = running_loss / dataset_sizes[phase]
                epoch_acc = running_corrects.double() / dataset_sizes[phase]
                
                if steps % 10 == 0:    
                    print('{} Loss: {:.4f} Accuracy: {:.4f}'.format(phase, epoch_loss, epoch_acc))
                
                # save best version of model
                if phase == VAL and epoch_acc > best_acc:
                    best_acc = epoch_acc
                    best_model_wts = copy.deepcopy(model.state_dict())
            
            print()
            
    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))
    print('Best val Accuracy: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

Building and training the classifier

Now that the data is ready, it's time to build and train the classifier. As usual, you should use one of the pretrained models from torchvision.models to get the image features. Build and train a new feed-forward classifier using those features.

We're going to leave this part up to you. Refer to the rubric for guidance on successfully completing this section. Things you'll need to do:

  • Load a pre-trained network (If you need a starting point, the VGG networks work great and are straightforward to use)
  • Define a new, untrained feed-forward network as a classifier, using ReLU activations and dropout
  • Train the classifier layers using backpropagation using the pre-trained network to get the features
  • Track the loss and accuracy on the validation set to determine the best hyperparameters

We've left a cell open for you below, but use as many as you need. Our advice is to break the problem up into smaller parts you can run separately. Check that each part is doing what you expect, then move on to the next. You'll likely find that as you work through each part, you'll need to go back and modify your previous code. This is totally normal!

When training make sure you're updating only the weights of the feed-forward network. You should be able to get the validation accuracy above 70% if you build everything right. Make sure to try different hyperparameters (learning rate, units in the classifier, epochs, etc) to find the best model. Save those hyperparameters to use as default values in the next part of the project.

One last important tip if you're using the workspace to run your code: To avoid having your workspace disconnect during the long-running tasks in this notebook, please read in the earlier page in this lesson called Intro to GPU Workspaces about Keeping Your Session Active. You'll want to include code from the workspace_utils.py module.

In [9]:
# TODO: Build and train your network -- Resnet
model = models.resnet50(pretrained=True)
print(model)
# Freeze layers
for param in model.parameters():
    param.requires_grad = False
# set in features
in_ftrs = model.fc.in_features
model.fc = nn.Linear(in_ftrs, len(class_names))
Downloading: "https://download.pytorch.org/models/resnet50-19c8e357.pth" to /root/.torch/models/resnet50-19c8e357.pth
100%|██████████| 102502400/102502400 [00:05<00:00, 20337666.33it/s]
ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential(
    (0): Bottleneck(
      (conv1): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (downsample): Sequential(
        (0): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): Bottleneck(
      (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
    )
    (2): Bottleneck(
      (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
    )
  )
  (layer2): Sequential(
    (0): Bottleneck(
      (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (downsample): Sequential(
        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): Bottleneck(
      (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
    )
    (2): Bottleneck(
      (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
    )
    (3): Bottleneck(
      (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
    )
  )
  (layer3): Sequential(
    (0): Bottleneck(
      (conv1): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (downsample): Sequential(
        (0): Conv2d(512, 1024, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): Bottleneck(
      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
    )
    (2): Bottleneck(
      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
    )
    (3): Bottleneck(
      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
    )
    (4): Bottleneck(
      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
    )
    (5): Bottleneck(
      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
    )
  )
  (layer4): Sequential(
    (0): Bottleneck(
      (conv1): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (downsample): Sequential(
        (0): Conv2d(1024, 2048, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): Bottleneck(
      (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
    )
    (2): Bottleneck(
      (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
    )
  )
  (avgpool): AvgPool2d(kernel_size=7, stride=1, padding=0)
  (fc): Linear(in_features=2048, out_features=1000, bias=True)
)
In [8]:
# TODO: Build and train your network -- VGG or Alexnet

# Load pretrained model
model = models.alexnet(pretrained=True)
print(model.classifier[6].out_features)

# Freeze layers
for param in model.features.parameters():
    param.requires_grad = False
    
# Replace last layer to match flower classes
num_features = model.classifier[6].in_features
features = list(model.classifier.children())[:-1]
features.extend([nn.Linear(num_features, len(class_names))])
model.classifier = nn.Sequential(*features)
print(model)
Downloading: "https://download.pytorch.org/models/alexnet-owt-4df8aa71.pth" to /root/.torch/models/alexnet-owt-4df8aa71.pth
100%|██████████| 244418560/244418560 [00:09<00:00, 25518369.00it/s]
1000
AlexNet(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))
    (1): ReLU(inplace)
    (2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (4): ReLU(inplace)
    (5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (6): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (7): ReLU(inplace)
    (8): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (9): ReLU(inplace)
    (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace)
    (12): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (classifier): Sequential(
    (0): Dropout(p=0.5)
    (1): Linear(in_features=9216, out_features=4096, bias=True)
    (2): ReLU(inplace)
    (3): Dropout(p=0.5)
    (4): Linear(in_features=4096, out_features=4096, bias=True)
    (5): ReLU(inplace)
    (6): Linear(in_features=4096, out_features=102, bias=True)
  )
)
In [9]:
# optimizer if Resnet
# optimizer_ft = optim.SGD(model.fc.parameters(), lr=0.001, momentum=0.9)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-9-bcd09cad1a29> in <module>()
      1 # optimizer if Resnet
----> 2 optimizer_ft = optim.SGD(model.fc.parameters(), lr=0.001, momentum=0.9)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    530                 return modules[name]
    531         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 532             type(self).__name__, name))
    533 
    534     def __setattr__(self, name, value):

AttributeError: 'AlexNet' object has no attribute 'fc'
In [10]:
# optimizer if VGG or Alexnet
optimizer_ft = optim.SGD(model.classifier.parameters(), lr=0.001, momentum=0.9)
In [11]:
# set loss function and dynamic learning rate
critereon = nn.CrossEntropyLoss()
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=9, gamma=0.1)
In [32]:
visualize_model(model)
Ground Truth:
<matplotlib.figure.Figure at 0x7f4c041da0b8>
Prediction:
In [14]:
model = train_model(model, critereon, optimizer_ft, exp_lr_scheduler, num_epochs=8)
Epoch 0/7
----------
train Loss: 0.2304 Accuracy: 0.0018
train Loss: 0.4285 Accuracy: 0.0108
train Loss: 0.5972 Accuracy: 0.0221
train Loss: 0.7505 Accuracy: 0.0375
train Loss: 0.8793 Accuracy: 0.0565
train Loss: 1.0099 Accuracy: 0.0748
train Loss: 1.1356 Accuracy: 0.0945
train Loss: 1.2467 Accuracy: 0.1169
train Loss: 1.3488 Accuracy: 0.1389
train Loss: 1.4530 Accuracy: 0.1624
train Loss: 1.5401 Accuracy: 0.1911
train Loss: 1.6361 Accuracy: 0.2157
train Loss: 1.7306 Accuracy: 0.2393
train Loss: 1.8126 Accuracy: 0.2669
train Loss: 1.8877 Accuracy: 0.2953
train Loss: 1.9632 Accuracy: 0.3249
train Loss: 2.0532 Accuracy: 0.3510
train Loss: 2.1285 Accuracy: 0.3805
train Loss: 2.2112 Accuracy: 0.4071
train Loss: 2.2916 Accuracy: 0.4338

valid Loss: 0.4176 Accuracy: 0.2738
valid Loss: 0.7770 Accuracy: 0.5623

Epoch 1/7
----------
train Loss: 0.0736 Accuracy: 0.0296
train Loss: 0.1437 Accuracy: 0.0598
train Loss: 0.2101 Accuracy: 0.0904
train Loss: 0.2778 Accuracy: 0.1215
train Loss: 0.3453 Accuracy: 0.1513
train Loss: 0.4116 Accuracy: 0.1819
train Loss: 0.4743 Accuracy: 0.2140
train Loss: 0.5443 Accuracy: 0.2447
train Loss: 0.6071 Accuracy: 0.2781
train Loss: 0.6722 Accuracy: 0.3101
train Loss: 0.7356 Accuracy: 0.3416
train Loss: 0.7995 Accuracy: 0.3738
train Loss: 0.8713 Accuracy: 0.4048
train Loss: 0.9399 Accuracy: 0.4344
train Loss: 1.0070 Accuracy: 0.4632
train Loss: 1.0696 Accuracy: 0.4963
train Loss: 1.1385 Accuracy: 0.5276
train Loss: 1.2052 Accuracy: 0.5577
train Loss: 1.2659 Accuracy: 0.5900
train Loss: 1.3219 Accuracy: 0.6247

valid Loss: 0.2949 Accuracy: 0.3117
valid Loss: 0.5759 Accuracy: 0.6271

Epoch 2/7
----------
train Loss: 0.0606 Accuracy: 0.0321
train Loss: 0.1151 Accuracy: 0.0665
train Loss: 0.1745 Accuracy: 0.0994
train Loss: 0.2342 Accuracy: 0.1313
train Loss: 0.2922 Accuracy: 0.1644
train Loss: 0.3483 Accuracy: 0.1973
train Loss: 0.4009 Accuracy: 0.2329
train Loss: 0.4615 Accuracy: 0.2666
train Loss: 0.5146 Accuracy: 0.3002
train Loss: 0.5664 Accuracy: 0.3362
train Loss: 0.6203 Accuracy: 0.3723
train Loss: 0.6742 Accuracy: 0.4052
train Loss: 0.7278 Accuracy: 0.4389
train Loss: 0.7738 Accuracy: 0.4751
train Loss: 0.8247 Accuracy: 0.5098
train Loss: 0.8761 Accuracy: 0.5444
train Loss: 0.9328 Accuracy: 0.5791
train Loss: 0.9873 Accuracy: 0.6137
train Loss: 1.0447 Accuracy: 0.6461
train Loss: 1.0911 Accuracy: 0.6824

valid Loss: 0.2705 Accuracy: 0.3105
valid Loss: 0.5331 Accuracy: 0.6406

Epoch 3/7
----------
train Loss: 0.0560 Accuracy: 0.0333
train Loss: 0.1031 Accuracy: 0.0693
train Loss: 0.1467 Accuracy: 0.1056
train Loss: 0.1950 Accuracy: 0.1409
train Loss: 0.2440 Accuracy: 0.1763
train Loss: 0.2958 Accuracy: 0.2105
train Loss: 0.3508 Accuracy: 0.2444
train Loss: 0.4022 Accuracy: 0.2790
train Loss: 0.4415 Accuracy: 0.3168
train Loss: 0.4840 Accuracy: 0.3536
train Loss: 0.5377 Accuracy: 0.3878
train Loss: 0.5958 Accuracy: 0.4219
train Loss: 0.6483 Accuracy: 0.4556
train Loss: 0.6978 Accuracy: 0.4908
train Loss: 0.7438 Accuracy: 0.5269
train Loss: 0.7911 Accuracy: 0.5626
train Loss: 0.8491 Accuracy: 0.5955
train Loss: 0.9087 Accuracy: 0.6284
train Loss: 0.9628 Accuracy: 0.6642
train Loss: 1.0107 Accuracy: 0.6999

valid Loss: 0.2077 Accuracy: 0.3423
valid Loss: 0.4047 Accuracy: 0.6797

Epoch 4/7
----------
train Loss: 0.0405 Accuracy: 0.0366
train Loss: 0.0887 Accuracy: 0.0726
train Loss: 0.1371 Accuracy: 0.1076
train Loss: 0.1800 Accuracy: 0.1439
train Loss: 0.2242 Accuracy: 0.1809
train Loss: 0.2720 Accuracy: 0.2163
train Loss: 0.3129 Accuracy: 0.2541
train Loss: 0.3567 Accuracy: 0.2906
train Loss: 0.3999 Accuracy: 0.3280
train Loss: 0.4440 Accuracy: 0.3649
train Loss: 0.4886 Accuracy: 0.4009
train Loss: 0.5258 Accuracy: 0.4386
train Loss: 0.5663 Accuracy: 0.4757
train Loss: 0.6185 Accuracy: 0.5098
train Loss: 0.6629 Accuracy: 0.5475
train Loss: 0.7126 Accuracy: 0.5817
train Loss: 0.7555 Accuracy: 0.6181
train Loss: 0.7987 Accuracy: 0.6555
train Loss: 0.8434 Accuracy: 0.6911
train Loss: 0.8978 Accuracy: 0.7245

valid Loss: 0.2217 Accuracy: 0.3191
valid Loss: 0.4092 Accuracy: 0.6626

Epoch 5/7
----------
train Loss: 0.0419 Accuracy: 0.0368
train Loss: 0.0802 Accuracy: 0.0748
train Loss: 0.1256 Accuracy: 0.1102
train Loss: 0.1681 Accuracy: 0.1473
train Loss: 0.2194 Accuracy: 0.1819
train Loss: 0.2582 Accuracy: 0.2199
train Loss: 0.3012 Accuracy: 0.2575
train Loss: 0.3435 Accuracy: 0.2947
train Loss: 0.3813 Accuracy: 0.3332
train Loss: 0.4229 Accuracy: 0.3701
train Loss: 0.4677 Accuracy: 0.4060
train Loss: 0.5075 Accuracy: 0.4425
train Loss: 0.5529 Accuracy: 0.4783
train Loss: 0.6032 Accuracy: 0.5147
train Loss: 0.6509 Accuracy: 0.5510
train Loss: 0.6943 Accuracy: 0.5867
train Loss: 0.7357 Accuracy: 0.6239
train Loss: 0.7802 Accuracy: 0.6600
train Loss: 0.8302 Accuracy: 0.6943
train Loss: 0.8707 Accuracy: 0.7317

valid Loss: 0.1956 Accuracy: 0.3325
valid Loss: 0.4136 Accuracy: 0.6614

Epoch 6/7
----------
train Loss: 0.0432 Accuracy: 0.0363
train Loss: 0.0803 Accuracy: 0.0746
train Loss: 0.1213 Accuracy: 0.1119
train Loss: 0.1628 Accuracy: 0.1490
train Loss: 0.2001 Accuracy: 0.1868
train Loss: 0.2366 Accuracy: 0.2253
train Loss: 0.2832 Accuracy: 0.2616
train Loss: 0.3264 Accuracy: 0.2988
train Loss: 0.3682 Accuracy: 0.3356
train Loss: 0.4092 Accuracy: 0.3724
train Loss: 0.4434 Accuracy: 0.4115
train Loss: 0.4861 Accuracy: 0.4495
train Loss: 0.5232 Accuracy: 0.4870
train Loss: 0.5723 Accuracy: 0.5221
train Loss: 0.6205 Accuracy: 0.5572
train Loss: 0.6552 Accuracy: 0.5952
train Loss: 0.6996 Accuracy: 0.6322
train Loss: 0.7404 Accuracy: 0.6705
train Loss: 0.7852 Accuracy: 0.7067
train Loss: 0.8328 Accuracy: 0.7425

valid Loss: 0.2130 Accuracy: 0.3313
valid Loss: 0.4141 Accuracy: 0.6553

Epoch 7/7
----------
train Loss: 0.0379 Accuracy: 0.0379
train Loss: 0.0746 Accuracy: 0.0763
train Loss: 0.1112 Accuracy: 0.1149
train Loss: 0.1474 Accuracy: 0.1537
train Loss: 0.1870 Accuracy: 0.1912
train Loss: 0.2263 Accuracy: 0.2289
train Loss: 0.2664 Accuracy: 0.2665
train Loss: 0.3077 Accuracy: 0.3025
train Loss: 0.3427 Accuracy: 0.3416
train Loss: 0.3831 Accuracy: 0.3794
train Loss: 0.4237 Accuracy: 0.4158
train Loss: 0.4652 Accuracy: 0.4530
train Loss: 0.5061 Accuracy: 0.4901
train Loss: 0.5469 Accuracy: 0.5264
train Loss: 0.5892 Accuracy: 0.5633
train Loss: 0.6262 Accuracy: 0.6021
train Loss: 0.6670 Accuracy: 0.6389
train Loss: 0.7141 Accuracy: 0.6749
train Loss: 0.7547 Accuracy: 0.7114
train Loss: 0.7955 Accuracy: 0.7492

valid Loss: 0.2114 Accuracy: 0.3362
valid Loss: 0.3676 Accuracy: 0.6846

Training complete in 11m 53s
Best val Accuracy: 0.877751
In [15]:
# save class to index mapping
model.class_to_idx = image_datasets[TRAIN].class_to_idx
In [18]:
# create checkpoint for rebuilding model
checkpoint = { 'state_dict': model.state_dict(), 'class_to_idx': model.class_to_idx, 'epochs': 8, 'arch': 'alexnet', 'optimizer': optimizer_ft.state_dict()  }
In [19]:
# update checkpoint - last model layers differ depending on pre-trained architecture
if checkpoint['arch'] == 'resnet50':
    checkpoint.update(fc = model.fc)
elif checkpoint['arch'] == 'vgg16' or checkpoint['arch'] == 'alexnet':
    checkpoint.update(classifier = model.classifier)
In [20]:
# TODO: Save the checkpoint 
torch.save(checkpoint, 'checkpoint.pth')
In [22]:
!ls
assets				   README.md
cat_to_name.json		   resnet50_checkpoint.pth
checkpoint.pt			   resnet_checkpoint.pt
flowers				   train.py
Image Classifier Project.ipynb	   vgg16_checkpoint.pth
Image Classifier Project-zh.ipynb  vgg_checkpoint.pth
LICENSE				   workspace-utils.py
predict.py

Loading the checkpoint

At this point it's good to write a function that can load a checkpoint and rebuild the model. That way you can come back to this project and keep working on it without having to retrain the network.

In [21]:
def rebuild_model(checkpoint_path):
  
    # load checkpoint from path
    checkpoint = torch.load(checkpoint_path)
    # rebuild pre-trained model from arch
    model = getattr(torchvision.models, checkpoint['arch'])(pretrained=True)

    if checkpoint['arch'] == 'vgg16' or checkpoint['arch'] == 'alexnet':  
        # freeze layers
        for param in model.features.parameters():
            param.requires_grad = False
        # set last layer    
        model.classifier = checkpoint['classifier']  

    # resnet version
    if checkpoint['arch'] == 'resnet50':
        # freeze layers
        for param in model.parameters():
            param.requires_grad = False
        # set last layer    
        model.fc = checkpoint['fc']    
    
    model.load_state_dict(checkpoint['state_dict'])
    model.optimizer = checkpoint['optimizer']
    model.class_to_idx = checkpoint['class_to_idx'] 

    return model
In [22]:
model = rebuild_model('checkpoint.pth')

Testing your network

It's good practice to test your trained network on test data, images the network has never seen either in training or validation. This will give you a good estimate for the model's performance on completely new images. Run the test images through the network and measure the accuracy, the same way you did validation. You should be able to reach around 70% accuracy on the test set if the model has been trained well.

In [23]:
# TODO: Do validation on the test set
def check_accuracy(dataloader):
    model.to('cuda')
    correct = 0
    total = 0
    with torch.no_grad():
        for inputs, labels in dataloader:
            inputs, labels = inputs.to('cuda'), labels.to('cuda')
            outputs = model(inputs)
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
    print('Accuracy: %d %%' % (100 * correct / total))     
In [24]:
# check validation set performance
check_accuracy(dataloaders[TEST])
Accuracy: 84 %

Save the checkpoint

Now that your network is trained, save the model so you can load it later for making predictions. You probably want to save other things such as the mapping of classes to indices which you get from one of the image datasets: image_datasets['train'].class_to_idx. You can attach this to the model as an attribute which makes inference easier later on.

model.class_to_idx = image_datasets['train'].class_to_idx

Remember that you'll want to completely rebuild the model later so you can use it for inference. Make sure to include any information you need in the checkpoint. If you want to load the model and keep training, you'll want to save the number of epochs as well as the optimizer state, optimizer.state_dict. You'll likely want to use this trained model in the next part of the project, so best to save it now.

In [16]:
print("Model: \n\n", model, '\n')
print("State dict keys: \n\n", model.state_dict().keys())
Model: 

 VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU(inplace)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU(inplace)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU(inplace)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU(inplace)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU(inplace)
    (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (18): ReLU(inplace)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU(inplace)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU(inplace)
    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (25): ReLU(inplace)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (27): ReLU(inplace)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU(inplace)
    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace)
    (2): Dropout(p=0.5)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace)
    (5): Dropout(p=0.5)
    (6): Linear(in_features=4096, out_features=102, bias=True)
  )
) 

State dict keys: 

 odict_keys(['features.0.weight', 'features.0.bias', 'features.2.weight', 'features.2.bias', 'features.5.weight', 'features.5.bias', 'features.7.weight', 'features.7.bias', 'features.10.weight', 'features.10.bias', 'features.12.weight', 'features.12.bias', 'features.14.weight', 'features.14.bias', 'features.17.weight', 'features.17.bias', 'features.19.weight', 'features.19.bias', 'features.21.weight', 'features.21.bias', 'features.24.weight', 'features.24.bias', 'features.26.weight', 'features.26.bias', 'features.28.weight', 'features.28.bias', 'classifier.0.weight', 'classifier.0.bias', 'classifier.3.weight', 'classifier.3.bias', 'classifier.6.weight', 'classifier.6.bias'])

Inference for classification

Now you'll write a function to use a trained network for inference. That is, you'll pass an image into the network and predict the class of the flower in the image. Write a function called predict that takes an image and a model, then returns the top $K$ most likely classes along with the probabilities. It should look like

probs, classes = predict(image_path, model)
print(probs)
print(classes)
> [ 0.01558163  0.01541934  0.01452626  0.01443549  0.01407339]
> ['70', '3', '45', '62', '55']

First you'll need to handle processing the input image such that it can be used in your network.

Image Preprocessing

You'll want to use PIL to load the image (documentation). It's best to write a function that preprocesses the image so it can be used as input for the model. This function should process the images in the same manner used for training.

First, resize the images where the shortest side is 256 pixels, keeping the aspect ratio. This can be done with the thumbnail or resize methods. Then you'll need to crop out the center 224x224 portion of the image.

Color channels of images are typically encoded as integers 0-255, but the model expected floats 0-1. You'll need to convert the values. It's easiest with a Numpy array, which you can get from a PIL image like so np_image = np.array(pil_image).

As before, the network expects the images to be normalized in a specific way. For the means, it's [0.485, 0.456, 0.406] and for the standard deviations [0.229, 0.224, 0.225]. You'll want to subtract the means from each color channel, then divide by the standard deviation.

And finally, PyTorch expects the color channel to be the first dimension but it's the third dimension in the PIL image and Numpy array. You can reorder dimensions using ndarray.transpose. The color channel needs to be first and retain the order of the other two dimensions.

In [19]:
# function used in first submission
def process_image_libs(image):
    ''' Scales, crops, and normalizes a PIL image for a PyTorch model,
        returns an Numpy array
    '''
    
    # TODO: Process a PIL image for use in a PyTorch model
    # open image
    img_pil = Image.open(image)
    # define transforms
    prep_image = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])
    ])
    # return transformed image
    return prep_image(img_pil)
In [25]:
# process image function using PIL and Python native functions
def process_image(image):
    ''' Scales, crops, and normalizes a PIL image for a PyTorch model,
        returns an Numpy array
    '''   
    # TODO: Process a PIL image for use in a PyTorch model
    # open image
    img = Image.open(image)
    # resize image
    if img.size[0] > img.size[1]:
        img.thumbnail((20000, 256))
    else:
        img.thumbnail((256, 20000))
    # crop image
    left = (img.width - 224) / 2
    bottom = (img.height - 224) / 2
    right = left + 224
    top = bottom + 224
    img = img.crop((left, bottom, right, top))
    # normalize
    img = np.array(img) / 255
    mean = np.array([0.485, 0.456, 0.406])
    std = np.array([0.229, 0.224, 0.225])
    img = (img - mean) / std
    # shift color channel for Pytorch
    img = img.transpose((2, 0, 1))
    # turn image into tensor
    img = torch.from_numpy(img).type(torch.FloatTensor)
    # return transformed image
    return img

To check your work, the function below converts a PyTorch tensor and displays it in the notebook. If your process_image function works, running the output through this function should return the original image (except for the cropped out portions).

In [30]:
def imgshow(image, ax=None, title=None):
    if ax is None:
        fig = plt.figure(figsize = [15,5])
    
    # PyTorch tensors assume the color channel is the first dimension
    # but matplotlib assumes is the third dimension
    image = image.transpose((1, 2, 0))
    
    # Undo preprocessing
    mean = np.array([0.485, 0.456, 0.406])
    std = np.array([0.229, 0.224, 0.225])
    image = std * image + mean
    
    # Image needs to be clipped between 0 and 1 or it looks like noise when displayed
    image = np.clip(image, 0, 1)
    
    
    plt.subplot(1,2,1)
    plt.axis('off')
    plt.imshow(image)
    
    return fig

Class Prediction

Once you can get images in the correct format, it's time to write a function for making predictions with your model. A common practice is to predict the top 5 or so (usually called top-$K$) most probable classes. You'll want to calculate the class probabilities then find the $K$ largest values.

To get the top $K$ largest values in a tensor use x.topk(k). This method returns both the highest k probabilities and the indices of those probabilities corresponding to the classes. You need to convert from these indices to the actual class labels using class_to_idx which hopefully you added to the model or from an ImageFolder you used to load the data (see here). Make sure to invert the dictionary so you get a mapping from index to class as well.

Again, this method should take a path to an image and a model checkpoint, then return the probabilities and classes.

probs, classes = predict(image_path, model)
print(probs)
print(classes)
> [ 0.01558163  0.01541934  0.01452626  0.01443549  0.01407339]
> ['70', '3', '45', '62', '55']
In [26]:
def predict(image_path, model, topk=5):
    ''' Predict the class (or classes) of an image using a trained deep learning model.
    '''
    # TODO: Implement the code to predict the class from an image file
    
    # process image
    model_image = process_image(image_path).unsqueeze_(0).cuda()
    
    # make prediction
    output = model(model_image)
    
    # convert to probabilities
    output = F.softmax(output, dim=1)
    
    # get 5 highest probabilities and classes
    outputs = list(torch.topk(output, topk))
    
    # convert probs tuple to numpy array, convert classes to list of strings
    probs = outputs[0].cpu().detach().numpy()[0]
    class_indices = outputs[1].cpu().detach().numpy()[0]
    
    # index to class conversion
    index_to_class = {val: key for key, val in model.class_to_idx.items()}
    top_classes = [index_to_class[each] for each in class_indices]
    
    return probs, [str(x) for x in list(top_classes)]

Sanity Checking

Now that you can use a trained model for predictions, check to make sure it makes sense. Even if the testing accuracy is high, it's always good to check that there aren't obvious bugs. Use matplotlib to plot the probabilities for the top 5 classes as a bar graph, along with the input image. It should look like this:

You can convert from the class integer encoding to actual flower names with the cat_to_name.json file (should have been loaded earlier in the notebook). To show a PyTorch tensor as an image, use the imshow function defined above.

In [27]:
# TODO: Display an image along with the top 5 classes
def display_preds(image, model=model):
    # make prediction
    probs, classes = predict(image, model)
    # map classes to tags
    tags = classes_not_numbers_list(classes, cat_to_name)
    # show image
    fig = imgshow(process_image(image).cpu().numpy(), ax=None, title=tags[0])
    fig.suptitle(tags[0], fontsize=16)

    # show top 5 classes
    
    # create subplot
    plt.subplot(1, 2, 2)
    # create horizontal barchart
    plt.barh(tags, probs, color='blue')
    # for dev purposes
    print(probs)
    print(tags)
    print(classes)
In [28]:
# set test image
test_image = 'flowers/test/10/image_07090.jpg'
In [31]:
display_preds(test_image)
[  9.98414516e-01   5.40446490e-04   5.28674223e-04   1.85980389e-04
   9.18743535e-05]
['globe thistle', 'artichoke', 'alpine sea holly', 'spear thistle', 'bee balm']
['10', '29', '35', '14', '92']
In [32]:
visualize_model(model)
Ground Truth:
<matplotlib.figure.Figure at 0x7fbae013a5f8>
Prediction: